Date: Thu, 19 Jun 2003 14:54:51 -0500
From: John Goerzen <jgoerzen@complete.org>
To: gopher@complete.org
Subject: [gopher] Re: Veronica-2 and robot exclusion
Message-ID: <20030619195451.GN6851@complete.org>
References: <009401c335b6$24e89c50$43da5982@killspy>
 <200306181744.KAA12996@floodgap.com>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <200306181744.KAA12996@floodgap.com>
User-Agent: Mutt/1.5.4i
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

On Wed, Jun 18, 2003 at 10:44:55AM -0700, Cameron Kaiser wrote:
> > Is this 'invisible' use of robots.txt allowed?
> 
> This would be the way I'd do it, and this is easy to do with a link list
> (like a Bucktooth gophermap file). There's probably no harm if the robots.txt
> file does get presented to the user, though.

It's pretty easy with UMN (and PyGopherd UMN module) as well:

Just add an entry in .names:

Path=./robots.txt
Type=X

-- John