.... Internet marketing resources, ecommerce web site design tutorials
  Taming the Beast - quality web marketing and ecommerce development services .... .


Return to web marketing and ecommerce articles index

Articles - Robots who Read

Before you submit your site to a search engine, you may want to consider what you want the search engine "bot" (the program that indexes your site) to "spider" (follow). You may have pages with sensitive information, or a scrap directory full of pages in progress that you would not like to see listed.

This can be achieved 2 ways. The first way is with a robots.txt file placed in the root directory of your web, but you must have full domain privileges in order for this to work. For further information, read our article on search engine spiders and robots.

- A quick note on the robots.txt file - do not leave it empty. This will indicate to some search engines that you do not want any part of the site indexed.

The other way to stop a number of bots from searching a page is to use META exclusion tags.

The following META tags can be used:


Putting this line between the <head> and </head> tags in your HTML  will prevent the bot from indexing that page.

An alternative is: 


The page will be indexed, but any hyperlinks in that page will not be spidered by the bot. 

Or a combination of the two:


Page will not be indexed, and other links will not be followed by the bot. This tag may also prevent some web-stripper (mirroring) software programs from downloading the page.

Further learning resources:

Optimization and Search engines

Search engine optimization tutorials. Increase your web site traffic by gaining better page rankings.

Meta Tags and Zen (revised)

Easy to understand tutorial on the use of Meta Tags for boosting search engine rankings. An optimized set of meta-tags is an important component of any overall search engine strategy.

Michael Bloch
Taming the Beast
Tutorials, web content, tools and software.
Web Marketing, Internet Development & Ecommerce Resources

Copyright information.... This article is free for reproduction but must be reproduced in its entirety with live links & this copyright statement must be included. Visit http://www.tamingthebeast.net  for free Internet marketing and web development articles, tutorials and tools! Subscribe for free to our popular ecommerce/web design ezine!

Click here to view article index 

Online meeting & webinar software review
Powerful, easy to use collaboration tools that can help improve your marketing sales and training efforts. Learn more about these services in this review & try a free trial!

The best shopping cart software
Our reviews of some of the best shopping carts around - free ecommerce solutions  through to premium services offering affiliate programs, marketing modules & online soft goods delivery.  Shopping cart software guide 

Autoresponder software/mailing list manager
 Read our beginners guide and reviews of all-in-one autoresponder & email marketing software solutions.

Credit card transaction fraud screening!  Effective fraud screening is an essential part of running an online businesses. Fraud transactions cost you money and can threaten your merchant account. Pick up a stack of transaction screening tips in this free guide! 

Need some advice/tools for writing/creating a web design, development or marketing proposal?






Get paid cash taking online surveys - free to join online 
survey companies that will pay you cash for your opinion!

In Loving Memory - Mignon Ann Bloch

copyright (c) 1999-2011  Taming the Beast  Adelaide - South Australia 

Profile - Contact - Privacy - Consultants Portfolio 

Search Site - Terms of Service - Social/environmental