Search Engine Facts
Search Engine Facts

Read our back issues

May 2017

December 2009

November 2009

October 2009

September 2009

August 2009

July 2009

June 2009

May 2009

April 2009

March 2009

February 2009

January 2009

December 2008

November 2008

October 2008

September 2008

August 2008

July 2008

June 2008

May 2008

April 2008

March 2008

February 2008

January 2008

December 2007

November 2007

October 2007

September 2007

August 2007

July 2007

June 2007

May 2007

April 2007

March 2007

February 2007

January 2007

December 2006

December 2006

November 2006

October 2006

September 2006

August 2006

July 2006

June 2006

May 2006

April 2006

March 2006

February 2006

Januray 2006

December 2005

November 2005

October 2005

September 2005

August 2005

July 2005

June 2005

May 2005

August 2005

March 2005

February 2005

January 2005

December 2004

November 2004

October 2004

September 2004

August 2004

July 2004

 

» Archive

 

SoftwareNerds.co.uk
All about software products and antivirus solutions.

e-Gear.dk
Good deals and offers on computers & hardware.

AVGDanmark.dk
AVG Antivirus offers top security solutions.

Home   Contact   Privacy policy    Partner sites

The importance of robots.txt

Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file.

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (it contains no HTML code). It must be placed in your root directory, for example "http://www.your-web-site.com/robots.txt".

How do I create a robots.txt file?

The robots.txt file is a simple text file. Open a simple text editor to create it. The content of a robots.txt file consists of so-called "records".

A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:

User-agent: googlebot
Disallow: /cgi-bin/

This robots.txt file would allow the "googlebot", which is the spider software program of Google, to retrieve every page from your site except for the files from the "cgi-bin" directory. All files in the "cgi-bin" directory will be ignored by googlebot.

The Disallow command works like a wild card. If you enter

User-agent: googlebot
Disallow: /logs

both "/logs.html" and "/logs/index.html" as well as all other files in the "logs" directory would not be indexed by search engines.

If you leave the Disallow line blank, you're telling the search engine that all files may be indexed. In any case, you must enter a Disallow line for every User-agent record.

There's much more to know about robots.txt files than we have space for in this newsletter. For that reason, we've created a special article about this topic on our Web site. Among other things, that article includes these topics:

  • Where to find user-agent names
  • 7 things you should avoid when designing your robots.txt
  • Tips and tricks for a good robots.txt file
  • Examples for easy and complex robots.txt files

http://www.Axandra.com/news/all-about-robots-txt.htm

Copyright Axandra.com - Internet marketing and search engine ranking software


Home   Contact   Privacy policy    Partner sites
January 2003 search engine articles