|
How your firewall can destroy
your search engine rankings
Some webmasters recently experienced a delisting from the
search engines without any good reason. The webmasters had
not done anything wrong and their web sites were optimized
for search engines. Nevertheless, the web sites had been removed
from search engines.
Poorly configured firewalls can block search engine
spiders
It turned out that the delisted web sites were all hosted
by the same hosting company. More precisely, the web sites
were all hosted by a hosting company that used a special firewall
software by SonicWALL Inc.
That firewall stopped the search engine spiders from accessing
their web sites. Google, Yahoo, MSN and all other search engines
that request the robots.txt
file couldn't index the web site anymore because the firewall
didn't allow that:
"An attacker could retrieve robots.txt from the
server, then use the contents of this file to discover the
path of an unprotected administration interface for the
server. The attacker may gain control of the webserver using
this interface.
The information gathered from robots.txt could be used
for system compromise and control of the web server."
(source)
This is the standard security settings of the SonicWALL firewall
and it basically means that your web site won't be spidered
by search engines if you use this firewall without customizing
it.
A firewall with these settings will drop the connection to
anyone requesting the robots.txt file so that it looks as
if the web site is offline. From an SEO point of view, this
is very bad for your web site because all good search engine
spiders request the robots.txt file before indexing your web
site.
What does this mean to you?
If your web site is not listed on search engines although
it has many good
incoming links and optimized
web page content, you should ask your web host if their
firewall blocks search engines that request the robots.txt
file. Your web host might not be aware of the problem.
Further information:
|
