Economic Times, Times Of India Block Google News, In.com

Published: May 27, 2008, 3:59 am

Update: Mohit Hira, Director of Marketing at Indiatimes, informs us that the company was in the midst of maintenance and upgrades, which required them to temporarily block all crawlers until today. These crawlers will be reinstated in phases today - as was the plan. Original Story: It appears that BCCL owned Times Internet Ltd, which operates the websites for India's largest English dailies the Economic Times, Times of India is preventing all web crawlers from accessing content from the websites. That means that sites like Google (NSDQ: GOOG) News, and indeed Google, Yahoo (NSDQ: YHOO) and the recently launched In.com aggregator from competitor Web18 will not be able to access and index news content from the Economic Times and Times of India. Sure enough, here's proof: the last Economic Times story indexed by Google News was six days ago - on 21st May 2008. Ditto for the Times of India. Crawlers follow a Robots Exclusion Protocol (or robots.txt protocol) which "is a

[ Full article ]

Blog link

Comments