"Noindex" and "Nofollow" made www.dynamsoft.com invisible in Google
A few months ago, our web team changed our site, www.dynamsoft.com, to make the visitor experience better.
Two weeks after the work was finished and the live web pages were finally updated, our web site team analyzed the web site usage data in Google Analytics. To our surprise, there were very few clicks from the organic Google content. The team tried searching in Google and almost none of our pages could be found. Even searching “Dynamsoft” or “SourceAnywhere” could not lead to our site.
Was it the magic of David Copperfield? Where had the website gone? If Google works properly (which we can assume it does), the causes of this issues are from the website. Our web team asked me for help. I reviewed the code of the web pages and found the cause of this issue. It was in a tag with the “robots” set to “noindex, nofollow”. -_- These tags are used when the pages are in our testing web server, to prevent search engines (such as Google, MSN and Yahoo) from indexing our not-ready web pages. After the pages are tested and before the pages are moved to our production server, the tags should be removed. But we did not do this.
So, we fixed the code and now finally the pages in www.dynamsoft.com can be searched in Google and other search engines again.
I also noticed several details but did not do further research:
- When www.dynamsoft.com was not searchable in Google, the site was still searchable in Yahoo and MSN. The reason could be that Yahoo and MSN do not update their search database as often as Google, or they just ignore the noindex and nofollow tag.
- When noindex and nofollow tags were used, the SSL encrypted pages, such as the download and purchase pages, were all still searchable.