Why Is My Website Not Indexed By Google, Yahoo Or Bing?
First and foremost I would like to iterate to you my dear readers that this article or post is intended for new bloggers with a newly created website. One of the most common questions I always see from most new bloggers is “My website is not indexed by Google, Yahoo and even on Bing after posting a few articles?” There are a couple of basic check-ups you could do to find out why is it no matter how you hard you put post-after-post on your website none of it is still on SERP (search engine result pages). I am setting aside the possible fact that you might have been experiencing an early effect of Google Panda since your website is still new – let’s be positive first. Also, I am setting aside SEO or search engine optimization) since the first problem that we need to solve here is search engine indexing. I have a list here that you can use to check your own blog site or website.
It’s true that search engines like Google Search, Yahoo, Bing and others alike have bots that crawls all over the Internet searching for contents, images, videos and many more and caching it into their servers. These are then served to all end-users looking for certain information when they typed for keywords on their search engine portals. BUT, if you wait for these bots to tell them that you have an existing article that might be useful for the readers without doing your part then it might take you “too” long before you see your blog site or website being indexed. Here some of the tips that you can do first.
1. Sitemap - This is the common forgotten file by most bloggers or webmasters new in building a blog site or website. A sitemap is an XML file that contains all the URLs or links available from your website and tells a particular search engine that your website is now ready for crawling. It’s like handling over all of the information available from your website to search engines like Google, Yahoo and Bing. If you are using WordPress platform for your website then there are only two plugins that I have already tried working like a charm. Although you can do it manually, I’d rather recommend you to utilize these WordPress plugins instead to save you more time.
(a). WordPress SEO by Yoast – this is the one I am using for this website. This plugin is an all-in-one plugin which will not only assist you with your SEO problems, indexation, RSS, permalinks, and many more. But the one you need here that I want to emphasize in relation to our topic is the XML Sitemap.
(b). Google XML Sitemaps – I’ve been using this one for years on my other websites and I find it very useful. The main feature of this plugin is to create a sitemap XML for you and notifies the search engines that your website is now ready to be crawled.
To configure any of these plugins is entirely up to you but there’s one thing that I want you to avoid when you set this up. Avoid as much as possible to include the following on your sitemap – categories, archive, and tags. Search engines like Google hates duplicate contents from SERP and this might lead you to being penalized or what we currently call “Panda-lized”. Another important thing that you have to take note is the sitemap link that will be generated from either one of the plugins I mentioned above.
Use only ONE of the plugins I recommended because it won’t do you any good having both of them – again I am only emphasizing here the sitemap features. Yoast have other features that you can use for your website.
2. Robots.txt – Now that your sitemap is ready then we will go to the next thing to check – your robots.txt. This is a small text (txt) file that is essential when you are building a website. Basically, this small text file is quite powerful when you learn to harness it. By default, when you install WordPress it uses a virtual robots.txt file on your website. But what we want to do is create our own customized robots.txt and to include the generated sitemap file from the first step. You can check my own robots.txt (link) and copy it BUT change the location and filename of your sitemap file. Open a notepad file (or any of your favorite text editor); copy my robots.txt and alter the location and filename of your sitemap. Notice the line from my robots.txt that contains:
Just change the URL and save it your file as “robots.txt” (without the quotes). Upload this on your website’s public_html directory or root directory. To check if you have successfully uploaded your file, open a browser and then type the URL on the address bar and check if you can view all those changes you’ve made.
I can recommend another WordPress plugin for you to use if you are not that confident creating your own robots.txt. Go and use iRobots.txt SEO, it is a plugin that will create a virtual SEO friendly robots.txt for you.
Now, we’re almost done. Here’s the last one you need to do and after this all you have to do is wait.
3. Submit your sitemap to Search Engines – although you already created a sitemap and configured your robots.txt, it’s still not enough. You may want to consider submitting your sitemap to search engines like Google and Bing. Yahoo Site Explorer is no longer available and they are now using Bing services to do this stuff. All you have to do is submit your sitemap on these search engines and you will have the chance as well how many links you have submitted and how many have been indexed.
You just need to login (register if you have no account yet) and submit your sitemap.
Basically, that’s all! Once Google, Yahoo, and Bing begins indexing your blog site or website then you need to take the next big step – SEO. You can either do it yourself or get some help specially if you are planning not only to get huge traffic on your website but also earn a lot of money from it. You can check SEO cheat sheet here.
If you have any further concern, don’t hesitate to shout.