Google have published a new entry in Google code offering information and advice on how to control crawling and indexing on your website…
How much do you know about the way Google crawls and indexes your website? In the new ‘Controlling Crawling and Indexing’ part of the Google Code website, you will find the answers to questions such as:
- Can I place the robots.txt file in a subdirectory?
- How do I specify AJAX-crawling URLs in the robots.txt file?
- I want to block a private folder. Can I prevent other people from reading my robots.txt file?
- I return 403 “Forbidden” for all URLs including the robots.txt file. Why is the site still being crawled?
- Can the robots meta tag be used to block a part of a page from being indexed?
Answers to these questions and a whole load more will prove very useful to new and existing site owners who want to understand the crawler. You can also learn about building SEO friendly websites and discover the solutions to current crawler and indexing problems you might be facing.
Despite his scampish image, Googlebot is a powerful automated tool that you can be utilised by optimising and effectively managing your robots.txt file, robots meta tag and X-Robots-Tag HTTP header element. This new offering from Google helps you do just that. There is even a discussion forum to help you find answers to questions that might not have been addressed.
Will you find this information useful?
- Search The Human Body With Google Body Browser - December 21, 2010
- Has This Site Been Hacked? - December 20, 2010
- How Do Twitter & Facebook’s 2010 Trends Compare With Google’s? - December 14, 2010
- Google Zeitgeist 2010: What Have You Been Searching For Planet Earth? - December 10, 2010
- Google Chrome Gets Store And Talks Chrome OS - December 8, 2010
- Google Issues Security Email For Website Optimiser - December 7, 2010
- Google Launches AdWords Global Market Finder - December 7, 2010
- New Android Phone Promises A Pure Experience - December 7, 2010
- Google AdWords Wants Your Products! - December 3, 2010
- Know How Mobile Internet Affects Your Website? Google Analytics does… - December 2, 2010
You may also be interested in:
- Is the Meta Description Tag an Endangered Species? Google says it reserves the right to change what description it shows about your webpage. Why do they do this? ...
- New Redesigned Webmaster Tools for Microsoft Bing Microsoft Completely Overhauls its Bing Webmaster Tools but sadly keeps our socks on... ...
- Better Ajax Support from Google Further developments from the SMX West conference as Google announce better search engine crawling for AJAX application on your site....
- How the Internet Works and Other Stories…by Google Read Google's lovely new book with lots of cool pictures and some great basic information on everything from: What is...
- Webmaster Videos – Speed Cloaking, Nofollow Internal Links and Browser Dependant Google SERPs! The Matt Cutts SEO surgery is open! ...
Cozy Digital is an