Re: [ga] DNSO.org web site and robots.txt

[William X Walsh <william@wxsoft.info>]

Saturday, May 18, 2002, 1:29:37 PM, George Kirikos wrote:

> Hello,

> I was curious as to why the DNSO website has a very restrictive
> robots.txt file:

> http://www.dnso.org/robots.txt

> With the top 3 lines of:

> User-agent: *
> Disallow: /constituency/work/
> Disallow: /clubpublic/

> it means that search engines like Google can't index the website
> (including mailing lists), and since the website itself lacks search
> capabilities, it's hard to find material (e.g. old posts) quickly.

> PROPOSAL: That the DNSO webmaster delete the current robot.txt file
> entirely.

> In this day and age with cheap bandwidth, I'm sure allowing the website
> to be indexed by Google and others wouldn't be costly, and has so many
> benefits.

I agree.  I've just changed the configuration of the dnspolicy search
indexer to ignore the robots.txt file on the dnso.org website.

-- 
Best regards,
William X Walsh <william@wxsoft.info>
--
Save Internet Radio!  
CARP will kill Webcasting!
http://www.saveinternetradio.org/

--
This message was passed to you via the ga@dnso.org list.
Send mail to majordomo@dnso.org to unsubscribe
("unsubscribe ga" in the body of the message).
Archives at http://www.dnso.org/archives.html