|
Google, Yahoo, the X-Robots directive and your
website rankings
Last week, Yahoo announced that they now support the X-Robots-Tag
in the HTTP header. This new tag allows you to influence
how Google and Yahoo index your website pages.
What is the X-Robots-Tag?
Google introduced the new X-Robots-Tag directive in 2007
to allow webmasters to control access to non-web page documents,
such as Adobe PDF files, video and audio files.
The
X-Robots-Tag is included in the HTTP
header of a document. The HTTP header is the initial
reply of a server to a query. It contains information about
the document that follows, including content type, creation
date, character set, encodings, etc.
The new X-Robots-Tag allows webmasters to add information
about search engine indexing to the HTTP header.
Which commands are supported by the X-Robots-Tag?
At this time, the X-Robots-Tag supports the following
commands:
- X-Robots-Tag: NOINDEX (Use this tag if you
don't want to show the URL in Google's search results)
- X-Robots-Tag: NOARCHIVE (Use this tag if you
don't want to see a Cache link in the search result pages
for the document)
- X-Robots-Tag: NOSNIPPET (Use this tag if you
don't want to display a summary in the search result
pages.)
- X-Robots-Tag: NOFOLLOW (Use this tag if you
don't want Googe and Yahoo to index the links in the
page.)
The X-Robots-Tag is currently supported by Google and
Yahoo. Other search engines don't support the tag yet.
How to add the X-Robots-Tag to your documents
It depends on your server and on the method you use to
create your web pages how you can add additional tags to
the HTTP header. If your web server uses Apache, you can
use the htaccess file to modify your HTTP headers.
In general, you shouldn't mess with the HTTP header if
you're not 100% sure what you're doing. A broken HTTP header
can keep search engines away so that no search engine will
index your website.
Instead of using the X-Robots-Tag, you can also add meta
tags with the same functionality to your web pages:
<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex">
You can edit these tags directly in the head part of your
web pages. They have the same effect as the corresponding
X-Robots-Tags explained above.
Do you need the X-Robots-Tag or the corresponding
meta tags?
If you don't want to limit access to your documents of
if you only have normal web pages on your website then
you don't need the X-Robots-Tag or the meta tags.
The X-Robots-Tag has been designed to restrict access
to your documents. That means that it does not help to
increase search engine rankings.
The new X-Robots-Tag and the meta robots tag allow you to
restrict access to your documents. If you want to make sure
that search engines index all of your web pages then you
have to optimize your
web pages.
Search engine robots must be able to easily parse your web
pages and they must be able to find the right elements on
your web pages. If you optimize the right web page elements
with the right keywords, your web page will get
high search engine rankings for these keywords.
|
