The Robots Meta Tag

The META Robots Tag was meant to provide users who cannot control the robots.txt file at their sites a last chance to keep their content out of search services. Since the whole idea of SEO is to get your content into the search engines (and well ranked!) it is unlikely you will ever use this tag and therefore you would normally omit it from your header.
The syntax should you require it is as follows:

<meta name=”robots” content=”comma separated list of terms”>
The comma separated list of terms and their meaning are as follows:
index means that robots are welcome to include this page in search services.
follow means that robots are welcome to follow links from this page to find other pages.
noindex allows the links to be explored even though the page is not indexed.
nofollow allows the page to be indexed but no links from the page are explored.
all means index and follow.
none means noindex and nofollow.
Here is an example that means index the page but restrict all robots from following links on the page.
<meta name=”robots” content=”nofollow”>
There is one variation that is obeyed by most robots that allows you to prevent the search engines from showing a “Cached” link for your site. You may want to do that if for example you have “buy it now” pages. If the page is cached but you have discontinued the offer users may still use the cached page to make purchases or as a bargaining tool to receive the items for less money. Also if you carry inventory information users may buy a product from the cached page even if it is now out of stock. It looks like this:
<meta name=”robots” content=”noarchive”>
Finally it is possible to instruct individual search engine bots by substituting “googlebot” or “msnbot” for “robots” but at the time of writing there is no Yahoo equivalent.
Update March 05, 2007
Using the robots meta tag on Google Webmaster Central Blog.
Yahoo! Search Support for ‘NOYDIR’ Meta Tags.
Meta RobotsTag 101.